{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4facdf7f-680e-4d28-908b-2b8408e2a741",
   "metadata": {},
   "source": [
    "# How to use multimodal prompts\n",
    "\n",
    "Here we demonstrate how to use prompt templates to format multimodal inputs to models. \n",
    "\n",
    "In this example we will ask a model to describe an image.\n",
    "\n",
    ":::info Prerequisites\n",
    "\n",
    "This guide assumes familiarity with the following concepts:\n",
    "\n",
    "- [Chat models](/docs/concepts/#chat-models)\n",
    "- [LangChain Tools](/docs/concepts/#tools)\n",
    "\n",
    ":::\n",
    "\n",
    "```{=mdx}\n",
    "import Npm2Yarn from \"@theme/Npm2Yarn\"\n",
    "\n",
    "<Npm2Yarn>\n",
    "  axios @langchain/core @langchain/openai\n",
    "</Npm2Yarn>\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "0d9fd81a-b7f0-445a-8e3d-cfc2d31fdd59",
   "metadata": {},
   "outputs": [],
   "source": [
    "import axios from \"axios\";\n",
    "\n",
    "const imageUrl = \"https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg\";\n",
    "const axiosRes = await axios.get(imageUrl, { responseType: \"arraybuffer\" });\n",
    "const base64 = btoa(\n",
    "  new Uint8Array(axiosRes.data).reduce(\n",
    "    (data, byte) => data + String.fromCharCode(byte),\n",
    "    ''\n",
    "  )\n",
    ");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "2671f995",
   "metadata": {},
   "outputs": [],
   "source": [
    "import { ChatPromptTemplate } from \"@langchain/core/prompts\";\n",
    "import { ChatOpenAI } from \"@langchain/openai\";\n",
    "\n",
    "const model = new ChatOpenAI({ model: \"gpt-4o\" })"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "4ee35e4f",
   "metadata": {},
   "outputs": [],
   "source": [
    "const prompt = ChatPromptTemplate.fromMessages(\n",
    "    [\n",
    "        [\"system\", \"Describe the image provided\"],\n",
    "        [\n",
    "            \"user\",\n",
    "            [{ type: \"image_url\", image_url: \"data:image/jpeg;base64,{base64}\" }],\n",
    "        ]\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "089f75c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "const chain = prompt.pipe(model);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "02744b06",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The image depicts a scenic outdoor landscape featuring a wooden boardwalk path extending forward through a large field of green grass and vegetation. On either side of the path, the grass is lush and vibrant, with a variety of bushes and low shrubs visible as well. The sky overhead is expansive and mostly clear, adorned with soft, wispy clouds, illuminated by the light giving a warm and serene ambiance. In the distant background, there are clusters of trees and additional foliage, suggesting a natural and tranquil setting, ideal for a peaceful walk or nature exploration.\n"
     ]
    }
   ],
   "source": [
    "const response = await chain.invoke({ base64 })\n",
    "console.log(response.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9b9ebf6",
   "metadata": {},
   "source": [
    "We can also pass in multiple images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "02190ee3",
   "metadata": {},
   "outputs": [],
   "source": [
    "const prompt = ChatPromptTemplate.fromMessages(\n",
    "    [\n",
    "        [\"system\", \"compare the two pictures provided\"],\n",
    "        [\n",
    "            \"user\",\n",
    "            [\n",
    "                {\n",
    "                    \"type\": \"image_url\",\n",
    "                    \"image_url\": \"data:image/jpeg;base64,{imageData1}\",\n",
    "                },\n",
    "                {\n",
    "                    \"type\": \"image_url\",\n",
    "                    \"image_url\": \"data:image/jpeg;base64,{imageData2}\",\n",
    "                },\n",
    "            ],\n",
    "        ],\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "42af057b",
   "metadata": {},
   "outputs": [],
   "source": [
    "const chain = prompt.pipe(model);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "513abe00",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The two images provided are identical. Both show a wooden boardwalk path extending into a grassy field under a blue sky with scattered clouds. The scenery includes green shrubs and trees in the background, with a bright and clear sky above.\n"
     ]
    }
   ],
   "source": [
    "const response = await chain.invoke({ imageData1: base64, imageData2: base64 })\n",
    "console.log(response.content)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Deno",
   "language": "typescript",
   "name": "deno"
  },
  "language_info": {
   "file_extension": ".ts",
   "mimetype": "text/x.typescript",
   "name": "typescript",
   "nb_converter": "script",
   "pygments_lexer": "typescript",
   "version": "5.3.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
